1. Field of the Invention
The invention relates to television signal processing systems and in particular to television signal processing systems for detecting selected segments of a television signal such as commercial advertisement segments.
2. Description of Related Art
It has long been a desire of many television viewers to have a system which automatically detects selected segments of a television signal such as commercial advertisements or other unwanted broadcast segments in real-time during a television broadcast for the purpose of, for example, muting the audio portion of the television broadcast during the unwanted broadcast material. Although a wide variety of techniques have been developed for detecting selected segments of television signals, heretofore there have been no sufficiently effective and economical systems that automatically operate in real-time for muting or similar purposes.
Examples of broadcast segment detection systems include systems for detecting commercial advertisements during the playback of a television program on a video cassette recorder (VCR) for the purpose of controlling the VCR to skip or scan over the commercial advertisements during playback such that a viewer may watch the recorded television program without significant commercial advertisement interruptions. One particularly effective system is described in U.S. Pat. No. 5,333,091 entitled xe2x80x9cMethod and Apparatus for Controlling a Videotape Player to Automatically Scan Past Recorded Commercial Messagesxe2x80x9d and sold under the trademark xe2x80x9cCommercial Advance.xe2x80x9d With the system of U.S. Pat. No. 5,333,091, an automatic editing device is coupled between a conventional VCR and a conventional television set. As a broadcast program is recorded on a videotape by the VCR, the editing device detects and records the timing and duration of selected events, such as black frames combined with low audio, that separate segments of the broadcast and creates a time-based map of the selected events. After the recording is completed, the editing device analyzes the time-based map and determines therefrom which segments are commercial advertisement segments and which are program segments. The editing device then records control signals on the videotape prior to and following the commercial advertisement segments. Upon playback, the control signals control the VCR to fast-scan over the commercial advertisement segments so that the recorded broadcast can be viewed substantially without interruption. Related techniques are described in U.S. Pat. No. 5,455,630 and in PCT application PCT/US94/00223 published as WO 94/16442.
Although the systems of U.S. Pat. Nos. 5,333,091 and 5,455,630 and PCT application PCT/US94/00223 are very effective in skipping over commercial advertisements and other unwanted broadcast material during the playback of a recorded broadcast program on a videotape, the systems do not operate in real-time during a received broadcast.
Other examples of broadcast segment detection systems include systems for detecting commercial advertisements during the recording of a television program on a VCR for the purposes of controlling the VCR to skip the commercial advertisements during recording such that a viewer may later watch the recorded television program without any commercial advertisement interruption whatsoever. An example is a system described in U.S. Pat. No. 4,602,297 entitled xe2x80x9cSystem for Editing Commercial Messages from Recorded Television Braodcastsxe2x80x9d which stores and delays the output of a received broadcast signal to a VCR until it can determine whether the delayed portion of the broadcast signal includes commercial advertisements. To this end, the system detects commercial advertisements or groups of commercial advertisement within the delayed portion by detecting amplitude drops in the broadcast signal and determining whether time intervals between the amplitude drops correspond to normal durations of commercial advertisements or groups of commercial advertisements. The system then outputs only those portions of the delayed signal to the VCR that do not correspond to commercial advertisements of groups of commercial advertisements. Hence, the broadcast signal is recorded without commercial advertisements for future playback.
Other examples of systems for detecting commercial advertisements during the recording of a television program are described in U.S. Pat. No. 4,750,052 entitled xe2x80x9cApparatus and Method for Automatically Deleting Selected Program Intervals from Recorded Television Broadcastsxe2x80x9d and U.S. Pat. No. 4,782,401 entitled xe2x80x9cEditing Method and Apparatus for Commercials During Video Recordingxe2x80x9d, both of which also operate to detect commercial advertisements based upon intervals occurring between amplitude drops or other fads in a received broadcast signal. Rather than storing and deleting the received signals prior to output to a VCR, however, the systems of U.S. Pat. Nos. 4,750,052 and 4,782,401 record the signals on a video tape using a VCR and control the VCR to rewind at the end of each commercial advertisement or group of advertisements such that further program signals are recorded over the commercial advertisements.
None of the aforementioned systems for detecting broadcast segments during the recording of a television program operate in real-time for the purpose of, for example, muting the audio during commercial advertisements.
Still other examples of broadcast segment detection systems include systems for detecting commercial advertisements in broadcast television signals for the purposes of verifying that selected commercial advertisements are being broadcast on the proper days and at the proper times. Examples include U.S. Pat. No. 5,504,518 entitled xe2x80x9cMethod and System for Recognition of Broadcast Segmentsxe2x80x9d wherein stored digitized segments of broadcast signals are processed by an expert system to determine whether any particular segment corresponds to a selected segment of interest, such as a selected commercial advertisement. Segments that cannot be identified by the expert system, such as new or otherwise unknown segments, are routed to a human operator for verifying that the segment represents a commercial advertisement rather than a portion of a television program. Selected segment information such as the date and time of broadcast of the segments of interest are recorded and tabulated in a report for use by, for example, broadcast advertisers for verifying that the selected segments were broadcast on the proper dates and at the proper times. As part of its analysis of the segments, the expert system compares broadcast signatures with a database of signatures. A variety of techniques are described for defining and detecting signatures. The expert system also considers various cues indicating the lengths and relative groupings of segments and the intervals therebetween. No real-time analysis appears to be performed by the expert system or by the human operator.
Another system for commercial advertisement broadcast verification is described in Japanese Laid-Open Patent Application No. 1-174192, of Sony Corporation, entitled xe2x80x9cTelevision Signal Processing Unitxe2x80x9d wherein an entire days worth of broadcast programs are recorded on videotape. The recorded broadcasts are analyzed to identify commercial advertisement segments by intervals therebetween as detected by luminance signal considerations. The commercial advertisements, and only the commercial advertisements, are then output and recorded on a second videotape. The second videotape thereby contains a days worth of commercial advertisements in extracted form that may be viewed by a human operator for broadcast verification purposes. Again, no real-time detection of commercial advertisements is performed.
Examples of systems for detecting broadcast segments, such as commercial advertisement segments, which have some real-time detection capability include U.S. Pat. Nos. 4,420,769 entitled xe2x80x9cDevice for the Automatic Editing, at the Receiver, of Unwanted Program Material from Broadcast Electrical Signalsxe2x80x9d and 4,750,213 entitled xe2x80x9cMethod and System for Editing Unwanted Program Material from Broadcast Signalsxe2x80x9d. U.S. Pat. No. 4,420,769 describes a system wherein a human operator identifies program segments to be eliminated from a broadcast signal causing a digital code word or signature for the segment to be stored in memory. Subsequently, if the same segment is encountered, the system automatically compares digitized portions of the received signal with the stored signatures to recognize the segment and then to blank out or otherwise eliminate the segment in real-time. One significant problem with this system is that a human operator must first detect program segments to be eliminated. Also, a significant amount of data processing is required to establish and compare the signatures in a reliable manner. U.S. Pat. No. 4,750,213, by the same inventor, sets forth improvements intended, in part, to reduce the processing time required. The system of U.S. Pat. No. 4,750,213 also includes a delay or other storage device for allowing a received signal to be delayed until a determination can be made by the human operator as to whether a particular segment of the received signal should be eliminated. If so, the received, delayed signal is output with the selected blanked out or otherwise eliminated. Although U.S. Pat. No. 4,750,213 may represent an improvement over U.S. Pat. No. 4,420,769, it shares the disadvantage that a human operator must detect program segments to be eliminated.
Another example of a system which has some real-time detection capability is U.S. Pat. No. 5,151,788 entitled xe2x80x9cMethod and Apparatus for Identifying and Eliminating Specific Material from Video Signalsxe2x80x9d wherein commercial advertisement broadcast segments are distinguished from program segments by the relative level of xe2x80x9cactivityxe2x80x9d within the beginning of each segment. If the activity level is high, the segment is presumed to be a commercial advertisement. The system detects the amount of activity within the first few seconds of a segment following a blank frame and operates to mute or otherwise eliminate the remaining portions of the segment if the detected activity level is representative of a commercial advertisement. The system may control a VCR to stop recording for the remainder of the segment. The patent describes that the optimum time period during which the activity level should be detected is 8 seconds. One disadvantage of the system is that the detection of commercial advertisements based upon an activity level is likely to be unreliable for many broadcasts and, even when the system does correctly detect a commercial advertisement, that detection does not occur until 8 seconds into the commercial advertisement.
Other references that relate to systems for detecting selected segments of a broadcast signal such as commercial advertisement segments of a television signal are described in the following references: U.S. Pat. No. 3,919,479 entitled xe2x80x9cBroadcast Signal Identification Systemxe2x80x9d; 4,229,765 entitled xe2x80x9cRemote Audio and Brightness Control System for a Televisionxe2x80x9d; 4,230,990 entitled xe2x80x9cBroadcast Program Identification Method and Systemxe2x80x9d; 4,259,689 entitled xe2x80x9cTelevision Advertising Editing Systemxe2x80x9d; 4,333,110 entitled xe2x80x9cTelevision Editing Systemxe2x80x9d; 4,390,904 entitled xe2x80x9cAutomatic Circuit and Method for Editing Commercial Messages from Television Signalsxe2x80x9d; 4,667,466 entitled xe2x80x9cBroadcast Program Identification Method and Apparatusxe2x80x9d; 4,697,209 entitled xe2x80x9cMethods and Apparatus for Automatically Identifying Programs Viewed or Recordedxe2x80x9d; 4,739,398 entitled xe2x80x9cMethod, Apparatus and System for Recognizing Broadcast Segmentsxe2x80x9d; 4,752,834 entitled xe2x80x9cReciprocating Recording Method and Apparatus for Controlling a Video Recorder so as to Edit Commercial Messages from a Recorded Television Signalxe2x80x9d; 5,019,899 entitled xe2x80x9cElectronic Data Encoding and Recognition Systemxe2x80x9d and European Patent Application Publication No. 0161512 entitled xe2x80x9cProgram Identification Systemxe2x80x9d.
Hence various techniques have been developed for detecting selected broadcast segments such as commercial advertisements within a television signal. None of the techniques described above, however, are effective for reliably and automatically detecting broadcast segments in real-time without the need of a human operator. According, it would be desirable to provide a simple, inexpensive system which reliably detects selected segments of a broadcast signal without requiring a human operator and wherein, either immediately or after some period of time, all or most selected segments can be detected in real-time.
In accordance with the invention, a method and apparatus, intended to be marketed under the mark Sound!Off(trademark), is provided for identifying selected segments of a broadcast signal in real-time for the purpose of, for example, muting the video and audio portions of the signal during the selected segment. Hereinafter, xe2x80x9cselected segmentxe2x80x9d refers to any selected portion of a broadcast signal including, for example, unwanted broadcast segments such a commercial advertisement segments. In one embodiment wherein the broadcast signal is a television signal, the method includes the steps of detecting a signature pattern associated with the beginning of a received segment of the television signal and comparing the signature pattern to stored signature patterns representative of selected segments. If the signature pattern matches one of the stored signature patterns, the received segment is thereby immediately identified as being one of the selected segments and can be processed accordingly in real-time to, for example, mute the audio and video portions of the television signal during the received segment as the television signal is output to, for example, a television monitor. If the signature pattern of the received segment does not match any of the stored signature patterns, the received segment is analyzed to determine whether it is nevertheless one of the selected segments and, if so, its signature pattern is stored along with the stored signature patterns. The analysis to determine whether the received segment is one of the selected segments even though its signature does not match any of the stored signatures may be performed by, for example, detecting the length of the received segment upon its completion and then determining whether the length of the received segment matches any of a pre-determined set of permissible lengths for selected segments. If so, the segment is identified as being a selected segment and its signature pattern is stored along with the other stored signature patterns such that, the next time the same segment is received, its signature will then match one of the stored signatures and therefore the received segment will be identified promptly at a selected segment and muted or otherwise altered in real-time. For an embodiment wherein the selected segments are commercial advertisement segments, the permissible lengths are, for example, 15 seconds, 30 seconds, and 60 seconds. The method may be performed by a television set or other device receiving a television signal such as a personal computer television (PCTV).
In one implementation, no stored signatures are initially provided. Hence, the first time any particular selected segment is encountered, it is not immediately identified as being a selected segment and therefore is not muted in real-time. However, the next time the same segment is encountered, the segment will be subject to immediate real-time muting because its signature will have been stored the first time it was received. Thereafter, as long as its signature remains stored, the segment will be subject to real-time muting whenever it is received. As more and more selected segments are encountered and their respective signature patterns stored, the more and more likely it becomes that any particular selected segment encountered while watching a television program will be promptly detected based upon its signature and muted in real-time. Hence, even though no stored signatures are initially provided, a database of signatures is soon developed. Also, in the exemplary implementation, the selected segment detection method may be performed by a television set even while the television set is otherwise not in use to thereby detect and record segment signatures in a background mode to quickly develop a database of signatures. Alternatively, a database of the signatures of common segments, such as common commercial advertisement segments, may be initially provided at the time of manufacture such that the common segments are thereby subject to immediate real-time muting even the first time received.
For many television broadcast systems, commercial advertisements appear in groups of two or more. When used in connection with such systems, the method of the invention preferably operates to not only mute or otherwise alter any detected commercial advertisements of a commercial advertisement group but also to mute or otherwise alter other unwanted broadcast segments appearing within the group, such as station breaks, news bulletins, public service announcements, etc. To this end, the method operates to identify the first of a group of commercial advertisements, then mutes or otherwise alters all segments during a predetermined minimum commercial advertisement group time period. The minimum group time period may be, for example, two minutes and twenty seconds. Hence, all segments appearing during the period are muted in real-time, including commercial advertisements for which the corresponding signature has not yet been stored (and which therefore would not otherwise be subject to immediate real-time muting) and segments that do not match the predetermined permissible lengths (and which therefore would not otherwise be subject to any muting whatsoever), such as the aforementioned station breaks, news bulletins, and public service announcements. Thus the method is not limited to muting only those segments that have matching stored signatures. Rather, the method operates to mute or otherwise alter all segments appearing in a detected commercial advertisement group. However, only the signatures of segments having lengths matching the permissible selected segment lengths are stored. The signatures of a non-commercial advertisement segment, such as a station break, news bulletin, or public service announcement, are stored only if the length of the segment happens to match one of the predetermined permissible segment lengths.
Depending upon the implementation, the method identifies the beginning of a received segment by detecting pre-determined event markers which separate segments. When implemented for use in North America, the method may, for example, identify a period of one or more black frames combined with low audio as being an event marker representative of the beginning of a segment. When implemented for use in Europe, the method may further identify periods of one or more colored frames, such as blue frames, combined with low audio as being an event marker. When implemented for use in Japan, the method may instead identify a frame break combined with a period of low audio as being an event marker. In any case, once an event marker denoting the beginning of a broadcast segment is detected, the method promptly detects a signature from the segment for comparison against stored signatures. The signature may, for example, be a binary hash code representative of whether selected lines of a selected frame near the beginning of the segment exceed pre-determined luminance threshold levels. By detecting the signature near the beginning of the segments, the signature can be compared against stored signatures to allow the segment to be muted or otherwise altered in real-time before much of the received segment has been displayed. Preferably, the signature is based on frames appearing during the first one second of the segment.